Papers with residual vector quantization
Towards Codec-LM Co-design for Neural Codec Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | Neural codec language models (or codec LMs) are emerging as a powerful framework for text-to-speech (TTS) despite the close interdependence of codecs and LM, research on codec and lms has largely remained siloed. |
| Approach: | They propose a frame-wise codec encoder that improves both LM log-likelihood and TTS metrics . they also propose LM codebook level dropout to efficiently navigate a portion of codec-LM design space . |
| Outcome: | The proposed codec-LM co-design improves intelligibility, audio quality and speaker control compared to a siloed baseline. |
PyramidCodec: Hierarchical Codec for Long-form Music Generation in Audio Domain (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to generate long music are inefficient and lack of structured representation. |
| Approach: | They propose a hierarchical discrete representation of audio for long audio-domain music generation using residual vector quantization on different levels of features. |
| Outcome: | The proposed method achieves competitive performance in terms of reconstruction quality and token per second (TPS) the proposed method facilitates training a language model that can generate well-structured long-form music for up to 3 minutes. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive language models excel in text-to-audio generation, but lag behind diffusion models by a non-trivial margin. |
| Approach: | They propose a framework that integrates multiple isolated transformers with causal conditioning and anti-causal alignment via reinforcement learning. |
| Outcome: | The proposed framework outperforms existing LM-based and diffusion-based systems in audio synthesis. |